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Abstract 

The  2006  EORTC  guidelines  for  erythropoietic  proteins  in  cancer-related  anemia  provide  the  most  up-to-date  assessment  of  the  evidence 
base.  Considering  general  concerns  in  medicine  about  clinicians’  adoption  of  evidence-based  guidelines,  it  is  critical  to  find  ways  of  bringing 
guidelines  to  the  point  of  care.  We  describe  the  rationale  behind  RESPOND,  a  web-based  clinical  guidance  system  based  on  the  EORTC 
guidelines,  and  the  methodologies  of  two  studies  conducted  to  validate  the  system.  In  a  first  descriptive  study,  experts  are  asked  to  rate  the 
accuracy  of  every  algorithm  derived  from  the  guidelines.  In  a  second  step  and  using  a  hybrid  matched  pre-post  design,  separate  retrospective 
and  prospective  patient  cohorts  matched  by  type  of  cancer  and  similarity  of  chemotherapy  regimen  (33  pairs)  are  used  to  examine  the  extent 
to  which  clinicians’  practice  patterns  converge  with  the  EORTC  guidelines  when  they  use  or  not  use  the  RESPOND  system.  It  is  hypothesized 
that  these  studies  will  provide  the  necessary  validation  for  RESPOND  as  an  evidence-based  clinical  support  tool. 

©  2007  Elsevier  Ireland  Ltd.  All  rights  reserved. 
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1.  Introduction 

Anemia  is  one  of  the  most  prevalent,  major  side  effects 
of  cancer  and  cancer  treatment  [1,2].  Attributed  to  such  fac¬ 
tors  as  inflammation-induced  changes  in  iron  homeostasis, 
production  of  erythroid  progenitor  cells,  erythropoietin  pro¬ 
duction,  and  the  life  span  of  red  blood  cells  [3]  as  well 
as  tumor  hypoxia  [4],  cancer-related  anemia  is  associated 
with  poor  prognosis  and  treatment  outcomes  [4-6]  and  has 
a  significant  impact  on  patients’  quality  of  life  [4,7].  In 
the  European  Cancer  Anaemia  Survey  (EC AS)  [2],  almost 
70%  of  cancer  patients  experienced  anemia  (hemoglobin 
(Hb)<  12g/dL)  during  a  6-month  period,  yet  only  39.8%  of 
them  were  treated:  17.4%  received  an  erythropoiesis  stimu¬ 
lating  agent  (ESA)  (either  alone  or  in  combination  with  blood 
transfusion  or  supplementary  iron),  14.9%  were  given  blood 
transfusion  alone,  and  6.5%  were  treated  with  iron  alone. 

Given  the  significant  therapeutic  benefits  of  ESAs  [8-10], 
but  also  the  variability  in  treatment  patterns  and  outcomes 
[1,2],  evidence-based  guidelines  have  been  proposed  by  the 
American  Society  of  Clinical  Oncology  (ASCO)  and  the 
American  Society  of  Hematology  (ASH)  [8]  and,  more 
recently,  by  the  European  Society  for  Research  and  Treat¬ 
ment  of  Cancer  (EORTC)  [9,10].  These  guidelines  provide 
evidence-based  direction  to  clinicians  in  the  (differential) 
diagnosis  and  management  of  anemia  in  cancer  patients. 

However,  there  is  significant  concern  in  medicine  at  large 
about  the  extent  to  which  evidence-based  guidelines  lead  to 
de  facto ,  evidence-based  decision-making  about  individual 
patient  care  [11].  Knowledge  about  and  adoption  of  guide¬ 
lines  by  clinicians  has  been  judged  -  intuitively  rather  than 
empirically  -  to  be  mixed  and  disappointing  [12],  and  has 
led  to  a  call  for  developing  strategies,  methods,  and  tools  to 
bring  evidence  closer  to  the  point  of  care. 

In  this  article,  we  describe  the  rationale  and  background 
behind  the  RESPOND  system,  as  well  as  the  methodology 


used  in  a  comprehensive  system  validation  effort.  RESPOND 
is  a  web-based  clinical  guidance  system  for  the  (differential) 
diagnosis  and  management  of  anemia  in  patients  with  cancer 
based  on  the  2006  Update  of  the  EORTC  guidelines  [10] 
and  intended  for  point-of-care  use  by  physicians,  nurses,  and 
other  cancer  clinicians. 


2.  Background 

2.1.  Management  of  cancer-related  anemia:  from 
evidence  to  guidelines  to  practice 

Erythropoietin  is  a  naturally  occurring  glycoprotein  that 
is  synthesized  mainly  in  the  kidneys  and  induces  prolifera¬ 
tion  and  differentiation  of  precursor  stem  cells  into  mature 
red  blood  cells.  The  human  erythropoietin  gene  was  cloned 
in  1985,  and  several  forms  of  the  recombinant  hormone 
(rHuEPO)  are  now  available  for  clinical  use.  These  erythro¬ 
poiesis  stimulating  agents  (ESAs)  are  potent  stimulators  of 
early  erythroid  progenitor  cells  in  the  bone  marrow  [13,14]. 

Treatment  with  ESAs  to  stimulate  erythropoiesis  and  to 
increase  Hb  levels  has  been  found  to  be  an  efficacious  and 
safe  way  to  manage  cancer-related  anemia  [8-10]  and  to 
improve  patients’  function  and  quality  of  life  [4].  Despite 
the  demonstrated  benefits  of  ESAs  in  controlled  studies, 
the  “real-world”  management  of  cancer-related  anemia  with 
ESAs,  as  documented,  for  instance,  in  the  2001  data  from 
the  EC  AS  study  [2],  is  suboptimal  and  characterized  by  con¬ 
siderable  variability  in  the  practice  patterns  of  prescribing 
physicians. 

By  setting  clinical  targets  and  providing  practice  rec¬ 
ommendations,  the  ASCO/ASH  [8]  and  EORTC  [9,10] 
guidelines  have  the  potential  to  decrease  this  variability  in 
practice  patterns  and  improve  patient  outcomes.  However,  the 
extent  to  which  physicians  are  familiar  with  these  guidelines 
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and  have  adopted  them  into  their  clinical  practice  is  unknown. 
If  other  conditions  are  any  indication,  this  adoption  should 
not  be  assumed. 

2.2.  Increasing  the  congruence  of  evidence -based 
guidelines  and  actual  clinical  practice  through  clinical 
guidance  systems 

One  method  to  increase  the  congruence  of  clinical  prac¬ 
tice  with  guidelines  is  to  design  computerized  systems  that 
provide  clinicians  with  evidence-based  guidance  based  on 
patient  data  entered  [15-17].  Many  decision  support  systems 
for  use  in  clinical  practice  have  been  developed  [18,19],  but 
few  have  been  tested  in  randomized  controlled  trials  (RCTs). 
The  most  effective  systems  are  those  that  are  integrated  into 
the  workflow,  provide  guidance  at  the  point  of  care,  and 
provide  actionable  recommendations  [20].  In  fact,  several 
systems  have  not  had  measurable  impacts  on  patient  out¬ 
comes  [20],  perhaps  because  of  inadequacies  in  such  factors 
as  development,  design,  user  interface,  validation,  acceptance 
by  clinicians,  and  integration  at  the  point  of  care.  Moreover, 
most  of  these  systems  are  not  evidence-based  but  knowledge- 
based  and  try  to  emulate  and  formalize  processes  of  clinical 
inference  [17]. 

In  an  initial  effort  to  make  the  ASH/ASCO  guidelines 
available  at  the  point  of  care  a  prototype  web-based  sys¬ 
tem  (ACT,  Anemia  Control  and  Testing)  was  developed  in 
2003-2004  under  the  auspices  of  Roche-Belgium.  In  this 
prototype,  Hb  (measured  by  a  point-of-care  testing  device 
[HemoCue,  Angelholm,  Sweden])  and  other  clinical  param¬ 
eters  were  analyzed  through  a  set  of  algorithms  derived  from 
the  ASH/ASCO  guidelines.  Recommendations  based  upon 
these  guidelines  were  then  generated.  With  the  publication  of 
the  EORTC  guidelines  in  2004  [9],  F.  Hoffmann-La  Roche 
AG  (Basel,  Switzerland)  proposed  to  adopt  the  prototype’s 
core  concept;  to  support  the  design,  development,  and  vali¬ 
dation  of  RESPOND,  a  web-based  clinical  guidance  system 
based  on  the  2006  Update  of  the  EORTC  guidelines  [10],  and 
to  make  this  system  available  to  cancer  clinicians  in  Europe, 
if  not  worldwide. 

From  the  onset,  cancer  anemia  experts  (Aapro,  Soubeyran, 
Van  Erps),  evidence-based  healthcare  researchers  (Abraham, 
MacDonald),  and  sponsors  (Turner,  Muenzberg,  Warrinnier) 
emphasized  the  need  for  rigorous  testing  and  validation  of 
the  RESPOND  system,  including  assessment  of  validity  and 
reliability — to  use  psychometric  terms  [21].  Remarkably, 
while  there  is  extensive  literature  on  clinical  software  val¬ 
idation  we  have  found  no  publications  providing  empirical 
evidence  for  the  validity  of  clinical  decision  support  sys¬ 
tems  driven  by  evidence-based  guidelines.  It  seems  as  if  the 
underlying  assumption  is  that  “if  the  algorithms  could  be 
programmed,  then  the  system  must  be  adequate”. 

Guidelines  like  those  proposed  by  the  EORTC  try  to  be 
as  complete  and  exhaustive  as  possible.  However,  the  fact 
that  some  evidence  may  be  of  lower  grades  of  strength,  or 
absent  altogether,  makes  that  even  the  best  guidelines  may 


have  incomplete  or  (as  of  yet)  inconclusive  recommenda¬ 
tions.  Consequently,  while  many  guidelines  may  translate 
rather  easily  into  programmable  algorithms,  others  require 
additional  interpretation  and  operationalization  before  they 
can  be  formalized  into  algorithms.  It  is  in  this  latter  process 
that  the  risk  for  incorrect  analysis,  erroneous  inference,  and 
inappropriate  clinical  guidance  is  the  highest. 

We  have  adopted  a  critical,  if  not  reluctant,  perspec¬ 
tive  on  computerized  systems  that  enable  clinicians  to  make 
evidence-based  clinical  decisions  at  the  point  of  care.  Intrin¬ 
sically,  we  are  committed  to  the  design  of  RESPOND  as  a 
web-based  decision  support  tool  to  provide  clinical  guidance 
to  cancer  clinicians  in  the  management  of  cancer-related  ane¬ 
mia.  On  the  other  hand,  we  oppose  having  the  guidelines 
programmed  into  a  computer  system  without  verification  and 
testing.  Therefore,  and  drawing  upon  principles  of  psychome¬ 
tric  theory  [21],  the  validation  of  RESPOND  is  comprised  of 
two  steps.  First,  a  group  of  experts  must  confirm  that  the  algo¬ 
rithms  correctly  translate  the  EORTC  guidelines  into  clinical 
recommendations.  Second,  RESPOND  must  be  tested  in  a 
clinical  setting  to  examine  the  extent  to  which  its  use  is  asso¬ 
ciated  with  clinical  practice  that  is  congruent  with  the  EORTC 
guidelines. 


3.  The  RESPOND  validation  methodology 

3.1.  Concept 

The  general  aim  of  this  validation  study  is  to  assess  vari¬ 
ous  dimensions  of  the  validity  of  RESPOND.  Thus  this  study 
aims  to  first  investigate  whether  RESPOND  correctly  trans¬ 
lates  the  EORTC  guidelines  into  clinical  recommendations. 
Once  this  has  been  established,  the  study  then  intends  to 
investigate  whether  the  use  of  RESPOND  is  associated  with 
practice  patterns  aligned  with  the  EORTC  guidelines. 

The  flowchart  in  Fig.  1  summarizes  the  process  of  this 
study: 

1.  Content  validity  and  accuracy  of  the  algorithms.  Are  the 
algorithms  embedded  in  RESPOND  accurate  operational¬ 
izations  of  the  assessment  and  management  rules  specified 
by  or  embedded  in  the  EORTC  guidelines? 

2.  Concurrent  validity  of  RESPOND.  To  what  extent  are 
clinicians’  anemia  care  practice  patterns  congruent  with 
EORTC  guidelines  when  they  are  using  the  patient  moni¬ 
toring  and  decision- support  functionality  of  RESPOND? 

3.  Discriminant  validity  of  RESPOND.  Are  the  anemia  care 
practice  patterns  of  clinicians  using  the  RESPOND  sys¬ 
tem  more  congruent  with  EORTC  guidelines  than  practice 
patterns  generated  in  the  absence  of  the  RESPOND  sys¬ 
tem?  In  other  words,  is  using  the  RESPOND  system 
associated  with  practice  patterns  that  are  more  congruent 
with  the  EORTC  guidelines  when  compared  to  prac¬ 
tice  patterns  observed  in  the  absence  of  the  RESPOND 
system? 
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Fig.  1.  Flowchart  of  the  study  objectives. 


3.2.  Study  populations 

3.2.1.  Overview 

For  Step  1:  physicians  and  nurses  with  an  established 
scientific  and  clinical  track  record  in  the  area  of  cancer- 
related  anemia;  including  members  of  the  EORTC  task  force 
that  developed  the  2006  revised  guidelines  and  experts  in 
evidence-based  healthcare. 

For  Step  2:  adult  cancer  patients  with  anemia  and 
undergoing  chemotherapy  matched  by  type  of  cancer  and 
chemotherapy  regimen.  Any  treatment  for  cancer,  anemia,  or 
other  clinical  condition  is  per  the  treating  physician’s  best 
clinical  judgment. 

There  are  no  required  treatment  protocols  for  this  study. 
All  prescribing  must  be  in  accordance  with  the  approved 
label/scientific  leaflet  in  Belgium. 

3.2.2.  Patient  selection  criteria  for  Step  2 

Patients  must  be  age  1 8  or  older,  diagnosed  with  a  solid  or 
hematological  malignancy,  treated  with  chemotherapy,  and 
anemic  (Hb  <  ll.Og/dL).  Patients  in  the  post-cohort  must 
provide  written  informed  consent.  Excluded  are  patients  who 
have  undergone  autologous  or  allogeneic  blood  stem  cell 
transplantation,  or  who  are  diagnosed  with  myelodysplastic 
syndrome. 

3.3.  Step  1:  Reliability  and  content  validity 

3.3.1.  Design 

The  first  step  of  this  study  employs  a  descriptive  design. 
A  purposively  selected  panel  of  experts  is  asked  to  rate  the 


accuracy  of  the  EORTC-based  algorithms  developed  for  the 
RESPOND  system.  It  is  assumed  that  if  a  given  algorithm  is 
judged  to  be  accurate,  RESPOND  will  not  only  feature  a  reli¬ 
able  analysis  model  for  a  given  guideline  but  also  adequately 
cover  the  content  of  that  guideline.  In  the  aggregate,  across 
guidelines,  it  will  permit  the  conclusion  that  the  RESPOND 
algorithms  reliably  and  validly  represent  the  EORTC  guide¬ 
lines. 

3.3.2.  Algorithm  evaluation  and  calculation  of  the 
intraclass  correlation  coefficient 

Experts  are  presented  by  email  with  a  document  listing 
each  EORTC  guideline,  the  algorithm(s)  operationalizing  a 
given  guideline,  the  required  data  elements,  and  the  guideline 
recommendation(s)  associated  with  each  outcome  of  a  given 
algorithm  (see  Table  1  for  an  example).  For  each  algorithm, 
experts  are  asked  to  indicate  with  a  binary  score  (yes/no) 
whether  the  algorithm  is  an  accurate  representation  of  the  ref¬ 
erent  EORTC  guideline.  For  any  “no”  answer,  the  expert  was 
asked  to  explain  why  and  to  offer  an  alternate  formulation. 

The  intraclass  correlation  coefficient  (ICC)  [22]  is  cal¬ 
culated  for  each  algorithm.  If  the  ICC  >  0.90,  the  algorithm 
will  be  considered  accepted.  Any  algorithm  with  ICC  <  0.90 
is  revised  per  the  suggestions  of  the  dissenting  experts  and 
resubmitted  by  email  to  the  panel  until  the  ICC  >0.90. 

In  the  event  that  after  3  email  voting  rounds  a  given  algo¬ 
rithm’s  ICC  is  still  <0.90,  the  algorithm  is  discussed  in  a 
conference  call  of  the  experts  in  order  to  achieve  consensus 
of  at  least  90%  of  the  experts. 

3.4.  Step  2:  Concurrent  and  discriminant  validity 

Once  the  reliability  and  content  validity  have  been  estab¬ 
lished,  it  is  important  to  examine  (1)  whether  clinicians’ 
practice  patterns  are  congruent  with  EORTC  guidelines 
when  they  use  RESPOND  (concurrent  validity)  and  (2) 
whether  clinicians’  practice  patterns  are  congruent  with 
EORTC  guidelines  when  using  RESPOND  versus  not  using 
RESPOND  (discriminant  validity). 

Especially  the  comparison  of  practice  patterns  when  using 
versus  not  using  RESPOND  is  methodologically  challenging. 
Note  that,  for  exploratory  purposes,  the  study  also  exam¬ 
ines  associations  between  the  use  of  RESPOND  and  clinical 
patient  characteristics,  and  between  congruence  scores  and 
clinical  patient  characteristics. 

This  part  of  the  study  is  being  done  in  a  mixed  oncology- 
hematology  center  in  a  non-university  teaching  hospital  in 
Belgium.  A  one-center  design  suffices  for  the  assessment  of 
concurrent  and  discriminant  validity.  Note  that  the  overall 
study  protocol  (steps  1  and  2)  was  approved  by  the  Ethi¬ 
cal  Committee  of  the  Algemeen  Stedelijk  Ziekenhuis  Aalst 
(Aalst,  Belgium),  where  the  study  is  being  conducted. 

3.4.1.  Design,  tool,  and  samples 

The  second  step  employs  a  hybrid  matched  pre-post 
design,  in  which  separate  pre  (retrospective)  and  post 
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Table  1 

Examples  of  selected  EORTC  guidelines,  proposed  algorithm,  and  expert  voting  procedures 


Guideline  2  Recommendation. 

In  cancer  patients  receiving  chemotherapy  and/or  radiotherapy,  treatment  with  erythropoietic  proteins  should  be  initiated  at  a  Hb  level  of  9-11  g/dL  based  on 
anaemia-related  symptoms  (grade  A). 


Set# 

Criteria  (if  met,  then  guideline  recommendation  provided) 

Accuracy  (Y/N) 

If  no,  then  please  indicate  your  suggestions  for  revision. 

2.1 

((if  chemo=yes)  AND/OR  (if  radio=yes))  AND 

(if  auto  transplant=no)  AND  (if  alio  transplant=no)  AND 

((if  Hb  >  8.9  g/dL)  AND  (if  Hb  <  1 1 .1  g/dL))  AND 

(if  symptoms=yes)  AND 

(ifepo=no) 

N  □  — ► 

Y  □ 

Guideline  5  Recommendation. 


Erythropoietic  proteins  may  be  considered  in  asymptomatic,  anaemic  patients  with  a  Hb  level  of  <11.9  g/dL  to  prevent  a  further  decline  in  Hb,  according  to 
indvidual  factors  (e.g.,  type/intensity  of  chemotherapy,  baseline  Hb)  and  the  duration  and  type  of  further  planned  treatment  (grade  B). 


Set# 

Criteria  (if  met,  then  guideline  recommendation  provided) 

Accuracy  (Y/N) 

If  no,  then  please  indicate  your  suggestions  for  revision. 

5.1 

(if  auto  transplant=no)  AND  (if  alio  transplant=no)  AND 
((if  Hb  >  8.9*  g/dL)  AND  (if  Hb  <  12.0  g/dL))  AND 
(if  symptoms=no)  AND 
(if  epo=no) 

N  □  — ► 

Y  □ 

* 

*  Note:  lower  Hb  limit  included  in  algorithm  since  Hb  <  9.0  g/dL  would  trigger  guideline  recommendation  #4. 


Guideline  6  Recommendation. 


We  do  not  recommend  the  prophylactic  use  of  erythropoietic  proteins  to  prevent  anaemia  in  patients  undergoing  chemotherapy  and/or  radiotherapy  who  have 
normal  Hb  values  at  the  start  of  treatment  (grade  A). 


Set# 

Criteria  (if  met,  then  guideline  recommendation  provided) 

Accuracy  (Y/N) 

If  no,  then  please  indicate  your  suggestions  for  revision. 

6.1 

((if  chemo=yes)  OR  (if  radio=yes))  AND 

(if  auto  transplant=no)  AND  (if  alio  transplant=no)  AND 

(if  Hb  >11.9  g/dL)  AND 

(if  gender  =  F) 

-<  z 

1 

6.2 

((if  chemo=yes)  OR  (if  radio=yes))  AND 

(if  auto  transplant=no)  AND  (if  alio  transplant=no)  AND 

(if  Hb  >13.9  g/dL)  AND 

(if  gender  =  M) 

N  □  — ► 

Y  □ 

1 

Respond  -  Patient  Data  Entry  -  Microsoft  Internet  Explorer  provided  by  Verizon  Online 
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Subject:  0922653 
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Visit  1:  Medications 

ESA  type 

ESA  route  of  administration 
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Frequency  of  ESA  administration 
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C  Epoetin  alFa 
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C  Other 

(•  Subcutaneous 
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IU 

(•lx  weekly 
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r  Q4W 

C  Other 
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Fig.  2.  Example  of  a  data  entry  screen  in  RESPOND. 
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Fig.  3.  Example  of  a  clinical  guidance  screen  in  RESPOND. 


(prospective)  matched  cohorts  are  established.  Patients  are 
matched  by  type  of  cancer  and  similarity  of  chemotherapy 
regimen. 

Clinicians  are  asked  to  use  the  system  prospectively  in 
the  care  of  a  sample  of  33  consenting  adult  patients  being 
treated  for  cancer  with  anemia  (see  Figs.  2  and  3  for  examples 
of  input  and  recommendation  screens).  This  sample  consti¬ 
tutes  the  prospective  post-cohort  (“post”  referring  to  “after” 
introduction  of  the  RESPOND  tool). 

Concurrent  with  the  prospective  component,  staff  review 
the  records  of  anemic  cancer  patients  who  underwent 
chemotherapy  at  the  center  between  4  and  12  months  prior 
to  the  introduction  of  the  RESPOND  tool  and  for  whom  data 
from  at  least  4  consecutive  visits  over  4  months  are  available. 
For  each  patient  recruited  into  the  post-cohort,  center  staff 
identifies  potential  patients  with  the  same  type  of  cancer  and 
similar  chemotherapy  regimen.  This  information  is  sent  to  an 
independent  investigator  for  matching.  A  total  of  33  patients 
will  thus  be  selected  to  constitute  the  pre-cohort  (“pre”  refer¬ 
ring  to  “prior  to”  introduction  of  the  RESPOND  tool). 

For  each  patient  in  the  study  (i.e.,  both  pre-  and  post¬ 
cohort  patients),  a  congruence  score  (CS)  is  calculated.  This 
score  quantifies  the  extent  to  which  the  care  provided  to  a 
given  patient  is  in  accordance  with  the  EORTC  guidelines. 


The  scores  of  the  post-cohort  will  be  used  to  assess  concurrent 
validity.  The  scores  of  the  matched  pre-  and  post-cohorts  will 
be  used  to  determine  discriminant  validity.  The  following 
schematic  summarizes  Step  2: 


2  matched  samples  (N=33  each): 

sample  1:  4  months  prospective 
data  after  RESPOND  (post) 

sample  2:  4  months  retrospective 
data  prior  to  RESPOND  (pre) 


Concurrent  validity: 

Congruence  scores  of  sample  1  (post) 

Discriminant  validity: 

Difference  between  sample  2  (pre)  and 
sample  1  (post)  congruence  scores 


Several  other  design  options  were  considered  but 
excluded.  Randomization  of  patients  within  the  same  center 
to  RESPOND-driven  versus  standard  care  is  likely  to  be  con¬ 
founded  by  within-clinician  spill-over  effects  of  knowledge, 
experience,  and  perspective.  Randomization  of  clinicians  or 
centers  to  RESPOND-driven  versus  standard  care  is  likely 
to  be  confounded  by  resentment  and  compensation  among 
those  clinicians  and  centers  randomized  to  standard  care.  This 
is  also  likely  to  be  the  case  in  a  clinician  or  center  cross¬ 
over  design  for  those  randomized  to  start  with  standard  care. 
Note  that  standard  care  does  not  preclude  knowledge  and 
application  of  EORTC  guidelines,  as  these  have  been  widely 
disseminated. 
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Table  2 

Calculation  of  the  congruence  score 

Item  Scoring 

Rule  out/treat  other  causes  of  anemia:  iron  deficiency  +0.25 

Rule  out/treat  other  causes  of  anemia:  bleeding  +0.25 

Rule  out/treat  other  causes  of  anemia:  nutritional  deficits  +0.25 

Rule  out/treat  other  causes  of  anemia:  hemolysis  +0.25 

ESA  treatment  initiated  at  Hb  range  9.0-1 1 .0  g/dL  +1 .00 

If  Hb  <  9.0  g/dL  patient  was  considered  for  blood  transfusion;  OR:  no  need  for  blood  transfusion.  +1 .00 

Anemic  asymptomatic  patient  with  Hb  <  1 1 .9  g/dL  was  considered  for  ESA  treatment  on  +1 .00 

individual  basis;  OR:  patient  does  not  fall  in  this  category 

Hb  target  set  at  Hb  range  12.0-13.0  g/dL  +1 .00 

Initial  ESA  dose  =  40,000  IU  epoetin  alfa  Q1W;  30,000  IU  epoetin  beta  Q1 W;  or  darbepoetin  alfa  +1 .00 

QlWor  Q3W 

Initial  ESA  dose  was  fixed  and  not  weight  adjusted  +1 .00 

Hb  target  achieved  at  4-8  weeks;  if  not,  individualized  dose  escalation  +1 .00 

Hb  levels  maintained  in  Hb  range  12.0-13.0  g/dL  for  up  to  8  weeks  (+0.125  per  week)  +1.00 

ESA  discontinued  if  Hb  >  13.0  g/dL  (or  Hb  in  range  12.0-13.0  g/dL  at  end  of  study  period)  +1 .00 

Maximum  possible  score  10.00 


3.4.2.  Data  model 1 

As  Step  2  is  an  observational  study,  only  those  data 
routinely  collected  (or  available  from  the  medical  record) 
are  entered  into  the  RESPOND  system.  Enrollment  data 
collected  include  eligibility  data,  patient  demographics,  can¬ 
cer  history,  anemia  history,  and  relevant  comorbidities.  At 
each  of  the  4  visits,  the  following  data  are  collected  as 
available:  weight,  hematological  parameters,  serum  ferritin, 
transferrin  saturation,  serum  creatinine,  serum  albumin,  C- 
reactive  protein,  ESA  treatment,  iron  supplementation,  blood 
transfusions,  cancer  treatment,  other  medications,  bleeding 
episodes,  signs  and  symptoms  of  anemia,  performance  status, 
and  signs  of  malnutrition. 

3.4.3.  Calculation  of  the  congruence  score 

The  congruence  score  (CS)  quantifies  the  extent  to  which  a 
patient’s  anemia  assessment,  management,  and  outcomes  are 
in  accordance  with  the  EORTC  guidelines.  It  was  developed 
by  a  subpanel  of  the  author  team  and  submitted  for  review 
and  evaluation  by  all  other  authors.  The  CS  reflects  the  10 
key  decisions  that  must  be  made  in  the  use  of  ES As  to  be  in 
accordance  with  the  EORTC  guidelines: 

1 .  Ruling  out  four  other  causes  of  anemia. 

2.  Initiating  ESA  treatment  at  Hb  range  9.0-1 1 .0  g/dL. 

3.  Considering/ruling  out  the  need  for  blood  transfusion. 

4.  Considering/ruling  out  the  possibility  of  asymptomatic 
presentation  of  anemia. 

5.  Setting  the  target  Hb  range  at  12.0-13.0  g/dL. 

6.  Using  the  correct  initial  ESA  dose. 

7.  Using  an  initial  fixed  dose,  not  a  weight-adjusted  dose. 

8.  Achieving  target  at  4-8  weeks;  otherwise,  initiate  an 
individualized  dose  escalation. 

9.  Maintaining  Hb  in  the  12.0-13.0  g/dL  range  for  up  to 
eight  weeks. 

10.  Discontinuing  ESA  treatment  if  Hb  >  13.0  g/dL. 


1  Complete  data  model  is  available  from  the  corresponding  author. 


For  each  key  decision  made  in  a  patient’s  care,  one  point 
is  given  for  range  of  0-10  total  points.  For  all  but  two  of  the 
key  decisions,  the  point  is  given  in  whole  units.  However,  for 
the  decision  regarding  other  causes  of  anemia  (decision  1), 
0.25  point  is  given  for  each  of  the  four  other  causes  ruled  out; 
i.e.,  ruling  out  all  four  generates  a  full  point.  Likewise,  for  the 
guideline  recommending  maintaining  Hb  in  range  for  up  to 
8  weeks,  0. 125  point  is  given  for  each  week  that  the  patient’s 
Hb  level  is  in  this  range. 

The  congruence  score  is  calculated  for  all  66  patients. 
Table  2  summarizes  the  CS  calculation. 

3.4.4.  Statistical  considerations 

The  sample  size  of  33  patients  in  each  cohort  (66  total) 
is  sufficient  for  detecting  mean  differences  of  0.50  standard¬ 
ized  units  under  the  non-central  t  distribution  and  correlation 
coefficients  of  0.45  with  power  of  0.80  at  alpha  of  0.05. 

Testing  the  concurrent  validity  of  RESPOND  will  be  done 
using  the  one- sample  /-test  to  examine  the  null  hypothesis  that 
the  CS  equals  10,  i.e.,  that  all  key  decisions  embedded  in  the 
EORTC  guidelines  have  been  made. 

Testing  for  statistical  significance  between  the  matched 
pre-  and  post-cohort  will  be  done  using  the  paired  /-test  (/- 
test  for  dependent  samples)  or,  in  the  case  of  violation  of 
statistical  assumptions,  its  nonparametric  analog.  This  test 
will  examine  whether  or  not  the  two  matched  subsamples 
show  statistically  significant  differences  in  the  CS.  This  is  a 
paired  test  and  the  basic  unit  of  calculation  is  the  standard¬ 
ized  difference  between  the  two  paired  patients  on  the  CS. 
The  null  hypothesis  to  be  tested  is  that  the  standardized  dif¬ 
ference  approximates  zero,  in  which  case  the  assumption  of 
discriminant  validity  will  be  rejected.  If  across  all  pairs  of 
matched  patients,  the  standardized  difference  exceeds  a  criti¬ 
cal  value,  it  will  be  possible  to  infer  the  discriminant  validity 
of  RESPOND. 

The  secondary  exploratory  analyses  on  the  potential 
association  between  the  use  of  RESPOND  and  clinical 
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patient  characteristics  and  the  association  between  con¬ 
gruence  scores  and  clinical  patient  characteristics  will  use 
measures  of  association.  Measures  will  be  selected  under 
consideration  of  the  levels  of  measurement  of  the  variables 
involved. 


4.  Discussion 

In  an  effort  to  develop  a  well-tested  and  validated  sys¬ 
tem,  the  RESPOND  initiative  merges  conventional  methods 
of  software  design  and  development  with  a  dual  scientific 
purpose  of  expert  evaluation  of  content  and  field  testing 
of  the  system  in  actual  patient  situations.  Certainly,  the 
psychometric-like  assessment,  evident  as  it  may  appear  to 
behavioral  and  social  scientists,  is  rather  unusual  in  health 
care.  Typically,  the  (implicit)  assumption  in  health  care  is  that, 
once  evidence-based  guidelines  have  passed  the  test  of  peer 
review  and  have  been  published  in  a  refereed  journal,  they 
are  ready  to  be  made  available  to  clinicians  through  various 
educational,  print,  or  electronic  means — and  that  improved 
patient  outcomes  will  follow. 

Step  1  has  been  completed  and  the  findings  are  reported 
elsewhere  [23].  Summarized,  consensus  was  achieved 
(ICC=1.00)  on  all  but  5  algorithmic  definitions  after  one 
round  of  review.  Following  revision,  the  remaining  5  defi¬ 
nitions  were  approved  unanimously  as  well  (ICC  =  1.00). 

The  major  focus  of  the  Step  2  component,  currently 
in  progress,  is  the  assessment  of  the  congruence  of  clini¬ 
cians’  practice  patterns  with  the  EORTC  guidelines  in  and 
between  two  subsamples  of  patients.  Admittedly,  the  impact 
of  RESPOND  on  patient  outcomes  is  not  (yet)  a  part  of  the 
RESPOND  initiative  and  clinicians  in  the  Step  2  process  are 
instructed  to  practice  according  to  their  best  clinical  judg¬ 
ment,  including  overriding  recommendations  generated  by 
RESPOND.  Examining  the  clinical  impact  of  RESPOND  is 
a  necessary  next  step,  preferably  in  the  form  of  a  cluster  ran¬ 
domized  controlled  trial  or,  alternately,  a  carefully  designed 
observational  study.  However,  we  believe  it  would  be  ethi¬ 
cally  questionable  to  subject  patients  to  an  intervention  (c.q., 
diagnosis  and  management  of  cancer-related  anemia  under 
the  guidance  of  a  decision-support  tool)  if  the  technology 
girding  the  intervention  has  not  been  thoroughly  tested  and 
validated.  Perhaps  a  stretch,  but  to  draw  an  analogy  with 
oncological  drug  development,  the  work  described  in  this 
article  is  comparable  to  the  discovery  phase  (conceptualiza¬ 
tion  of  the  system),  pre-clinical  testing  and  proof  of  concept 
(reliability  and  content  validity  assessment),  initial  formu¬ 
lation  (programming),  and  safety  and  (clinician)  tolerability 
testing  using  patients  (Step  1). 

Part  of  the  need  for  testing  and  validation  is  also  safety 
(and  by  extension  liability)  driven.  In  the  end,  a  system  like 
RESPOND  may  provide  clinical  advice  that  is  followed  by 
clinicians  and  made  part  of  patient’s  therapeutic  plan.  As 
with  any  intervention  (or,  for  that  matter,  medical  device), 
the  necessary  precautions  and  actions  must  have  been  taken 


to  assure  that  the  system  is  safe — even  before  it  may  be  found 
to  be  clinically  effective. 

There  are  several  additional  challenges  that  will  need  to 
be  addressed  as  RESPOND  evolves  into  a  clinically  useful 
support  tool.  First,  the  knowledge  regarding  the  use  of  ESAs 
in  cancer  patients  with  anemia  is  likely  to  evolve.  Guidelines 
in  place  today  are  likely  to  change  as  new  evidence  about  the 
effectiveness  and  safety  of  ESAs  in  supportive  cancer  care 
becomes  available.  RESPOND,  like  systems  incorporating 
evidence-based  guidelines  in  other  areas  of  healthcare,  will 
need  to  be  adapted  as  the  evidence  base  changes.  It  will  be 
impractical  to  resubmit  RESPOND  to  a  complete  validation 
exercise  as  described  in  this  article — nor  is  that  necessary. 
Critical  and  essential  will  be  to  replicate  Step  1 :  the  verifi¬ 
cation  by  experts  that  an  algorithm  is  an  accurate  and  valid 
representation  of  the  referent  guideline. 

A  second  challenge  concerns  the  extent  to  which  a 
guideline-based  system  like  RESPOND  reflects  the  com¬ 
plexity  of  clinical  decision-making  in  the  management  of 
cancer-related  anemia  and  the  clinical  dynamics  of  comor¬ 
bidities  in  diagnosis,  management,  and  prevention — and  the 
clinical  outcomes  that  can  be  achieved.  Unfortunately,  this 
is  where  the  evidence-based  healthcare  model  runs  into  a 
“self-imposed”  limitation  by  attaching  a  higher  grade  of 
relevance  to  randomized  controlled  trials  (and  any  ensuing 
meta- analyses)  than  to  other  research  models.  RCTs  give 
the  most  reliable  and  internally  valid  information  about  the 
efficacy  of  a  treatment  (i.e.,  outcomes  attributable  to  the 
intervention  under  investigation)  by  controlling  the  influence 
of  extraneous  variables  through  such  sequential  procedures 
as  specifying  inclusion/exclusion  criteria,  randomization  to 
treatment  arms,  and  post  hoc  statistical  control  of  residual 
confounds.  Typically,  RCTs  enroll  relatively  homogeneous 
groups  of  patients  with  limited  variability  in  potential  con¬ 
founding  variables.  Comorbidities  tend  to  be  considered 
potential  confounds  and  by  and  large  are  managed  through 
exclusion — with  only  the  more  common  ones  being  included. 
Consequently,  comorbidities  are  less  likely  to  be  addressed 
in  evidence-based  guidelines;  in  contrast  to  the  reality  of 
daily  clinical  practice  where  patients  are  heterogeneous  and 
treatment  effectiveness  may  be  confounded.  The  RESPOND 
system  addresses  this  issue  by  allowing  clinicians  to  override 
the  guideline-recommended  approach  and  documenting  the 
reason  for  doing  so. 

Relatedly,  a  third  challenge  concerns  the  consideration 
of  patient  preferences  in  planning  care.  Patients  choose 
between  treatment  options  based  on  their  personal  evaluation 
of  attributes,  consequences,  and  the  physical,  psychological, 
and  financials  burdens  involved.  In  RCTs,  the  building  blocks 
of  evidence-based  guidelines,  one  implicitly  assumes  that 
patients  have  expressed  their  treatment  preferences  by  pro¬ 
viding  informed  consent  to  be  included  in  the  study.  Actual 
patient  preferences  are  not  investigated  in  efficacy  studies, 
even  though  later  “real  world”  effectiveness  may  be  affected 
by  the  extent  to  which  patients  agree  to,  accept,  tolerate,  and 
refuse  the  recommended  treatment.  As  a  guideline-driven 
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tool,  RESPOND  cannot  accommodate  patient  preferences. 
However,  being  an  advisory  rather  than  a  prescriptive  tool, 
clinicians  should  feel  comfortable  overriding  recommen¬ 
dations  if  these  are  contrary  to  patient  preferences,  and 
document  the  reasons  why. 

As  the  evidence-based  guidelines  movement  is  rapidly 
achieving  a  stage  of  maturity,  in  which  major  guidelines  are 
developed  for  wide  dissemination  and  to  guide  local  guide¬ 
lines,  one  can  expect  more  systems  like  RESPOND  to  be 
developed.  Important  functional  characteristics  of  such  sys¬ 
tems  have  already  been  proposed  [17],  however  guidance  as 
to  the  scientific  validation  of  systems  remains  lacking.  It  is 
hoped  that  the  methodology  described  here  can  be  of  assis¬ 
tance  to  other  clinical  areas.  In  the  meanwhile,  the  RESPOND 
initiative,  complex  as  it  is,  should  provide  the  necessary  evi¬ 
dence  that  the  RESPOND  system  is  based  on  algorithms  that 
accurately  specify  the  EORTC  guidelines,  and  that  it  assists 
clinicians  in  caring  safely  for  cancer  patients  with  anemia  in 
accordance  with  these  guidelines. 
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